{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ZTc_CmMKFn3B"
      },
      "source": [
        "# Monitoring LLMs with LangChain, OpenAI, LangKit, and WhyLabs \n",
        "\n",
        "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/whylabs/langkit/blob/main/langkit/examples/Langchain_OpenAI_LLM_Monitoring_with_WhyLabs.ipynb)\n",
        "\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "nWOcAHe_GueX"
      },
      "source": [
        "We'll show how you can generate out-of-the-box text metrics using LangKit + Langchain and monitor them in the WhyLabs Observability Platform.\n",
        "\n",
        "You'll need a free WhyLabs account & and OpenAI API account to follow along.\n",
        "\n",
        "With LangKit, you'll be able to extract and monitor relevant signals from unstructured text data, such as:\n",
        "\n",
        "- [Text Quality](https://github.com/whylabs/langkit/blob/main/langkit/docs/features/quality.md)\n",
        "- [Text Relevance](https://github.com/whylabs/langkit/blob/main/langkit/docs/features/relevance.md)\n",
        "- [Security and Privacy](https://github.com/whylabs/langkit/blob/main/langkit/docs/features/security.md)\n",
        "- [Sentiment and Toxicity](https://github.com/whylabs/langkit/blob/main/langkit/docs/features/sentiment.md)\n",
        "\n",
        "For this example, we'll pay attention to sentiment change between prompts and responses. Sentiment can be a valuable metric to understand how users interact with your LLM in production and how any system prompts or template updates change responses.\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "l8lA6efWsWn3"
      },
      "source": [
        "### Install LangKit & LangChain\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "68hb_GdNFn3C"
      },
      "outputs": [],
      "source": [
        "%pip install langkit[all]==0.0.2\n",
        "%pip install langchain==0.0.205"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "QdXoUXPfi3gs"
      },
      "source": [
        "### Set OpenAI and WhyLabs credentials:\n",
        "To send LangKit profiles to WhyLabs we will need three pieces of information:\n",
        "\n",
        "- API token\n",
        "- Organization ID\n",
        "- Dataset ID (or model-id)\n",
        "\n",
        "Go to [https://whylabs.ai/free](https://whylabs.ai/free) and grab a free account. You can follow along with the quick start examples or skip them if you'd like to follow this example immediately.\n",
        "\n",
        "1. Create a new project and note its ID (if it's a model project, it will look like `model-xxxx`)\n",
        "2. Create an API token from the \"Access Tokens\" tab\n",
        "3. Copy your org ID from the same \"Access Tokens\" tab\n",
        "\n",
        "Get your OpenAI API key from your [OpenAI account](https://openai.com/)\n",
        "\n",
        "Replace the placeholder string values with your own OpenAI and WhyLabs API Keys below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "YwRkou_9dsjD"
      },
      "outputs": [],
      "source": [
        "# Set OpenAI and WhyLabs credentials\n",
        "import os\n",
        "\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"OPENAIAPIKEY\"\n",
        "os.environ[\"WHYLABS_DEFAULT_ORG_ID\"] = \"WHYLABSORGID\"\n",
        "os.environ[\"WHYLABS_DEFAULT_DATASET_ID\"] = \"MODELID\"\n",
        "os.environ[\"WHYLABS_API_KEY\"] = \"WHYLABSAPIKEY\""
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "E6jXPMnZBVWt"
      },
      "source": [
        "#### Import LangChain callbacks, OpenAI LLM, and additional language metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "jz1HPE-edslZ"
      },
      "outputs": [],
      "source": [
        "from langchain.callbacks import WhyLabsCallbackHandler\n",
        "from langchain.llms import OpenAI\n",
        "\n",
        "# Import additional language metrics\n",
        "import langkit.sentiment\n",
        "import langkit.topics"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "zM0o1X_tBWYT"
      },
      "source": [
        "#### Initialize WhyLabs callback and OpenAI GPT models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "t7X1tXZadsnv"
      },
      "outputs": [],
      "source": [
        "# Initialize WhyLabs Callback & GPT model with LangChain\n",
        "whylabs = WhyLabsCallbackHandler.from_params()\n",
        "llm = OpenAI(temperature=0, callbacks=[whylabs])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "syw5ewmbBXdJ"
      },
      "source": [
        "#### Generate responses on prompts & close WhyLabs session\n",
        "\n",
        "The rolling logger for whylogs will write profiles every 5 minutes or when `.flush()` or `.close()` is called."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q4h1qpMFdsqc",
        "outputId": "99c8815a-ce4e-4f7e-b6c5-69d06e7b7792"
      },
      "outputs": [],
      "source": [
        "# generate responses to positive prompts from LLM\n",
        "result = llm.generate(\n",
        "    [\n",
        "        \"I love nature, its beautilful and amazing!\",\n",
        "        \"This product is awesome. I really enjoy it.\",\n",
        "        \"Chatting with you has been a great experience! you're very helpful.\"\n",
        "    ]\n",
        ")\n",
        "print(result)\n",
        "\n",
        "# close WhyLabs Session which will also push profiles to WhyLabs\n",
        "whylabs.close()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "9Y9aF2QkJbHR"
      },
      "source": [
        "Thats it! Language metrics about the prompts and model responses are now being tracked in WhyLabs.\n",
        "\n",
        "Navigate to the profile tab and click on \"View details\" over the `prompt.sentiment_nltk` metric to see the distribution of sentiment scores for the prompt. \n",
        "\n",
        "In this example, all the prompts have a positive sentiment score of 80+.\n",
        "\n",
        "![](../../static/img/langchain-positive-sentiment.png)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Click on the \"Show Insights\" button to see further insights about language metrics for prompts and responses.\n",
        "\n",
        "![](../../static/img/langchain-insights.png)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "5VELa2bFSlDh"
      },
      "source": [
        "As more profiles are written on different dates, you'll get a time series pattern you can analyze & set monitors like in the [Demo org](https://bit.ly/3NOq0Od).\n",
        "\n",
        "You can also backfill batches of data by overwriting the date and time as seen in [this example](https://github.com/whylabs/langkit/blob/main/langkit/examples/Batch_to_Whylabs.ipynb)."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "qQMxbWh_RPSD"
      },
      "source": [
        "![](../../static/img/sentiment-monitor.png)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "x8yvZ6FfJlK6"
      },
      "source": [
        "### Watch the sentiment value change from negative prompts\n",
        "After inspecting the results in WhyLabs, try changing your prompts to trigger a change in the metric you're monitoring, such as prompt sentiment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gMgy7vEYGKmM"
      },
      "outputs": [],
      "source": [
        "# Intialize WhyLabs Callback & GPT with Langchain\n",
        "whylabs = WhyLabsCallbackHandler.from_params()\n",
        "llm = OpenAI(temperature=0, callbacks=[whylabs])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cHrQTHfBf-5C",
        "outputId": "ef6cb1c9-2334-4430-ebb6-61ecbf62a33c"
      },
      "outputs": [],
      "source": [
        "result = llm.generate(\n",
        "    [\n",
        "        \"I hate nature, its ugly.\",\n",
        "        \"This product is bad. I hate it.\",\n",
        "        \"Chatting with you has been a terrible experience!.\"\n",
        "        \"I'm terrible at saving money, can you give me advice?\"\n",
        "    ]\n",
        ")\n",
        "print(result)\n",
        "\n",
        "# close WhyLabs Session\n",
        "whylabs.close()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "c8S03ALAVvwO"
      },
      "source": [
        "Viewing the histogram results in WhyLabs again, you can see the sentiment value change from only positive prompts to containing a range of negative & positive prompts.\n",
        "\n",
        "We can configure monitors to alert us automatically when the sentiment value changes in the monitor manager tab.\n",
        "\n",
        "![](../../static/img/langchain-negative-sentiment.png)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "lyzF4hCFROqX"
      },
      "source": [
        "In this example, we've seen how you can use LangKit to extract and monitor sentiment from unstructured text data. You can also use LangKit to extract and monitor other relevant signals from text data."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "7e685se4aPE0"
      },
      "source": [
        "# More Resources\n",
        "\n",
        "Learn more about monitoring LLMs in production with LangKit\n",
        "\n",
        "- [Intro to LangKit Example](https://github.com/whylabs/langkit/blob/main/langkit/examples/Intro_to_Langkit.ipynb)\n",
        "- [LangKit GitHub](https://github.com/whylabs/langkit)\n",
        "- [whylogs GitHub - data logging & AI telemetry](https://github.com/whylabs/whylogs)\n",
        "- [WhyLabs - Safeguard your Large Language Models](https://whylabs.ai/safeguard-large-language-models)\n",
        "\n",
        "\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "whylabs-textmetricstoolkit-EeFODeF5-py3.8",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.2"
    },
    "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
